fix(dataprotection): terminate cluster restores through resource owners by leon-ape · Pull Request #10777 · apecloud/kubeblocks

leon-ape · 2026-08-13T09:11:21Z

Problem

Cluster or Component deletion is the restore cancellation signal. The target PVC deletion timestamp is not. Cancellation must preserve controller ownership: each controller deletes only the objects it created and removes only the finalizer it added.

Changes

Cluster restore protection

add a restore-specific Cluster finalizer owned only by ClusterRestoreReconciler
add it while Cluster restore is active or PVC-scoped DP resources remain
retain protection for Restore=False while the live Cluster still carries initial-restore intent
on Cluster deletion, observe cached Restore/PVC state and wait for resource owners to finish cleanup
treat every internal Restore, including Completed and Failed objects, as pending cleanup while Cluster deletion is in progress
never delete Restore/helper PVC objects or patch target PVC finalizers from ClusterRestoreReconciler

VolumePopulator termination

treat Cluster deletion as Cluster-wide cancellation and Component deletion as Component-scoped cancellation
ignore target PVC deletion as a business signal
delete VP-owned execution and postReady Restore objects, then wait for RestoreReconciler
delete the VP-owned helper PVC and wait for disappearance
finally remove only the VP-owned target PVC DP finalizer
continue teardown for retained PVCs using identity recorded before restore resources are created
verify exact UID/owner identity before every destructive operation

Restore creation ordering

wait for the Cluster restore-protection finalizer before entering the PVC restore state machine
before creating a postReady Restore, use the controller cache to confirm that the Cluster is not deleting and still carries restore protection
stop postReady creation when the target Component is deleting
use watches and normal requeue for dependency progression; no direct API reader is used as a correctness mechanism

Identity

propagate Cluster UID through Cluster-owned VCT restore intent and VP-owned DP labels
validate the live PVC → Instance/InstanceSet → Component → Cluster owner chain before restore starts
record the verified Cluster and Component UID on a new target PVC before VP creates helper/Restore resources
do not migrate or adopt resources from older controller versions in this deletion-focused PR

Ownership invariants

ClusterRestoreReconciler only patches its Cluster restore-protection finalizer
VolumePopulator never adds/removes the Cluster finalizer and never deletes target PVC/Component/Cluster
RestoreReconciler remains the sole owner of Restore Job/Pod cleanup and Restore finalizers
App/InstanceSet remains the sole deletion path for target PVCs

Dependencies

fix(dataprotection): ignore target PVC deletion during restore #10819, fix(dataprotection): watch VolumePopulator restore dependencies #10825, and fix(workloads): aggregate restore status for instance API #10831 are merged into main
this branch is synchronized with the latest main containing those changes

Tests

full controllers/dataprotection package
Cluster restore-intent UID injection/cleanup tests
cmd/dataprotection compile test
ownership, ordered termination, Retain owner detachment, failed/terminal Restore protection, target deletion neutrality, postReady creation guards, and UID mismatch coverage

Fixes #10755

apecloud-bot · 2026-08-13T09:11:32Z

Auto Cherry-pick Instructions

Usage:
  - /nopick: Not auto cherry-pick when PR merged.
  - /pick: release-x.x [release-x.x]: Auto cherry-pick to the specified branch when PR merged.

Example:
  - /nopick
  - /pick release-1.1

CLA Recheck Instructions

Usage:
  - /recheck-cla: Trigger a re-check of CLA status for this pull request.
Example:
  - /recheck-cla

codecov · 2026-08-13T09:29:09Z

Codecov Report

❌ Patch coverage is 67.76699% with 166 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.79%. Comparing base (63373ca) to head (c77f21c).

Files with missing lines	Patch %	Lines
...llers/dataprotection/volumepopulator_controller.go	63.86%	102 Missing and 44 partials ⚠️
...llers/dataprotection/cluster_restore_controller.go	81.48%	13 Missing and 7 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10777      +/-   ##
==========================================
+ Coverage   65.76%   65.79%   +0.02%     
==========================================
  Files         510      511       +1     
  Lines       64958    65455     +497     
==========================================
+ Hits        42721    43067     +346     
- Misses      18449    18555     +106     
- Partials     3788     3833      +45

Flag	Coverage Δ
unittests	`65.79% <67.76%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…restore-finalizer

…/10755-cluster-restore-finalizer # Conflicts: # controllers/dataprotection/volumepopulator_controller.go

…restore-finalizer # Conflicts: # controllers/dataprotection/volumepopulator_controller.go # controllers/dataprotection/volumepopulator_controller_test.go

…restore-finalizer

…restore-finalizer # Conflicts: # controllers/dataprotection/volumepopulator_controller.go # controllers/dataprotection/volumepopulator_controller_test.go

fix: protect clusters during restore cleanup

471bdbf

leon-ape added the nopick Not auto cherry-pick when PR merged label Aug 13, 2026

github-actions Bot added the size/L Denotes a PR that changes 100-499 lines. label Aug 13, 2026

leon-ape mentioned this pull request Aug 13, 2026

fix: validate restore volume binding lifecycle #10765

Closed

chore: update restore controller RBAC

bfd8698

leon-ape changed the title ~~fix: protect Cluster lifecycle during restore execution~~ fix(dataprotection): protect Cluster lifecycle during restore execution Aug 13, 2026

leon-ape marked this pull request as ready for review August 13, 2026 10:01

leon-ape requested review from a team, ldming and wangyelei as code owners August 13, 2026 10:01

leon-ape added 2 commits August 14, 2026 11:21

Merge remote-tracking branch 'origin/main' into bugfix/10755-cluster-…

334c375

…restore-finalizer

fix: harden cluster restore cleanup ownership

443884d

github-actions Bot added size/XL Denotes a PR that changes 500-999 lines. and removed size/L Denotes a PR that changes 100-499 lines. labels Aug 14, 2026

leon-ape added 2 commits August 14, 2026 11:38

fix: carry cluster UID in restore intent

9c2b9f9

fix: serialize cluster restore teardown

0a7fbfd

github-actions Bot added size/XXL Denotes a PR that changes 1000+ lines. and removed size/XL Denotes a PR that changes 500-999 lines. labels Aug 14, 2026

leon-ape added 5 commits August 14, 2026 16:05

Merge remote-tracking branch 'origin/main' into bugfix/10755-cluster-…

c62a35c

…restore-finalizer

fix: adopt legacy cluster restore resources

8ff769e

fix: retain protection for legacy restore orphans

b39098a

Merge remote-tracking branch 'origin/main' into bugfix/10755-cluster-…

34b3901

…restore-finalizer

fix: harden legacy restore adoption races

c189ea9

leon-ape mentioned this pull request Aug 28, 2026

fix: clean up deleting restore PVCs #10605

Closed

leon-ape added 2 commits August 28, 2026 16:53

Merge remote-tracking branch 'origin/main' into bugfix/10755-cluster-…

2e92af1

…restore-finalizer

fix(dataprotection): block direct target PVC cleanup

e9d4922

leon-ape mentioned this pull request Aug 28, 2026

fix(dataprotection): ignore target PVC deletion during restore #10819

Merged

fix(dataprotection): clarify restore wait on target deletion

5a87e69

leon-ape added 6 commits August 31, 2026 15:02

Merge dependency watches into cluster restore protection

f4b49db

refactor: keep volume populator dependencies together

e0a4147

Merge VolumePopulator dependency organization

158bb50

Merge origin/main into VolumePopulator dependency watches

6538347

Merge latest VolumePopulator dependency watches

1cceea0

fix: terminate restores through resource owners

f14dc86

leon-ape changed the title ~~fix(dataprotection): protect Cluster lifecycle during restore execution~~ fix(dataprotection): terminate Cluster restores through resource owners Aug 31, 2026

leon-ape changed the base branch from main to bugfix/volume-populator-dependency-watches August 31, 2026 07:34

leon-ape added 5 commits August 31, 2026 15:42

fix(dataprotection): use conventional error messages

619a213

fix(dataprotection): cover cross-component restore dependencies

032ec05

Merge branch 'bugfix/volume-populator-dependency-watches' into bugfix…

de20ccd

…/10755-cluster-restore-finalizer # Conflicts: # controllers/dataprotection/volumepopulator_controller.go

fix(dataprotection): fan out post-ready restore events

6b64196

Merge branch 'bugfix/volume-populator-dependency-watches' into bugfix…

4045c7a

…/10755-cluster-restore-finalizer # Conflicts: # controllers/dataprotection/volumepopulator_controller.go

leon-ape changed the title ~~fix(dataprotection): terminate Cluster restores through resource owners~~ fix(dataprotection): terminate cluster restores through resource owners Aug 31, 2026

fix(dataprotection): scope post-ready termination by owner

3f035b2

Base automatically changed from bugfix/volume-populator-dependency-watches to main August 31, 2026 08:25

Merge remote-tracking branch 'origin/main' into bugfix/10755-cluster-…

a8070a9

…restore-finalizer # Conflicts: # controllers/dataprotection/volumepopulator_controller.go # controllers/dataprotection/volumepopulator_controller_test.go

leon-ape mentioned this pull request Aug 31, 2026

fix(workloads): aggregate restore status for instance API #10831

Merged

leon-ape added 12 commits August 31, 2026 17:41

fix(dataprotection): finish restore teardown after retain

6b390ed

fix(dataprotection): close restore protection race

c16b9c2

fix(dataprotection): continue retained restore by identity

248cd2e

refactor(dataprotection): keep parent deletion explicit

0d3a1e0

fix(dataprotection): fail closed on component replacement

243d4b8

Merge remote-tracking branch 'origin/main' into bugfix/10755-cluster-…

cd18bd6

…restore-finalizer

fix(dataprotection): simplify restore deletion coordination

9be4daf

test(dataprotection): align cluster restore test filename

f6d90c1

Merge remote-tracking branch 'origin/main' into bugfix/10755-cluster-…

dd7d019

…restore-finalizer

fix(dataprotection): tighten restore deletion identity

ca8a901

refactor(dataprotection): clarify restore lifecycle coordination

7992763

Merge remote-tracking branch 'origin/main' into bugfix/10755-cluster-…

c77f21c

…restore-finalizer # Conflicts: # controllers/dataprotection/volumepopulator_controller.go # controllers/dataprotection/volumepopulator_controller_test.go

Navigation Menu

Uh oh!

fix(dataprotection): terminate cluster restores through resource owners - #10777

fix(dataprotection): terminate cluster restores through resource owners#10777
leon-ape wants to merge 46 commits into
mainfrom
bugfix/10755-cluster-restore-finalizer

leon-ape commented Aug 13, 2026 •

edited

Loading

Uh oh!

apecloud-bot commented Aug 13, 2026

Uh oh!

codecov Bot commented Aug 13, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

leon-ape commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

Cluster restore protection

VolumePopulator termination

Restore creation ordering

Identity

Ownership invariants

Dependencies

Tests

Uh oh!

apecloud-bot commented Aug 13, 2026

Uh oh!

codecov Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

leon-ape commented Aug 13, 2026 •

edited

Loading

codecov Bot commented Aug 13, 2026 •

edited

Loading